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Abstract 

A deluge of empirical research became available on MOOCs in 2013-2015 and this research is available in 
disparate sources. This paper addresses a number of gaps in the scholarly understanding of MOOCs and 
presents a comprehensive picture of the literature by examining the geographic distribution, publication 
outlets, citations, data collection and analysis methods, and research strands of empirical research 
focusing on MOOCs during this time period. Results demonstrate that (a) more than 80% of this 
literature is published by individuals whose home institutions are in North America and Europe, (b) a 
select few papers are widely cited while nearly half of the papers are cited zero times, and (c) researchers 
have favored a quantitative if not positivist approach to the conduct of MOOC research, preferring the 
collection of data via surveys and automated methods. While some interpretive research was conducted 
on MOOCs in this time period, it was often basic and it was the minority of studies that were informed by 
methods traditionally associated with qualitative research (e.g., interviews, observations, and focus 
groups). Analysis shows that there is limited research reported on instructor-related topics, and that even 
though researchers have attempted to identify and classify learners into various groupings, very little 
research examines the experiences of learner subpopulations. 
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Introduction 

The term Massive Open Online Course (MOOC) describes an evolving ecosystem of online learning 
environments that encompass a spectrum of course designs (Rodriguez, 2012). Between 2013 and 2015 a 
deluge of empirical research became available on the topic, and this research is available in disparate 



sources ranging from professional journals in a variety of disciplines, through annual conference 
proceedings, to the proceedings of newly-developed conferences and workshops focusing specifically on 
MOOCs (e.g., the Learning at Scale conferences). While the MOOC phenomenon has been subject to 
numerous interpretations in the mass media (Kovanovic, Joksimovic, Gasevic, Siemens, & Hatala, 2015; 
Selwyn, Bulfin, & Pangrazio, 2015) and some analyses of the the state of the field have been published 
(e.g., Bayne & Ross, 2014; Ebben & Murphy, 2014), researchers currently lack a systematic synthesis of 
the empirical literature published on the topic. A collective research effort is required to fully understand 
the impact of MOOCs (Reich, 2015), and researchers will benefit from analyses and syntheses of the 
literature, especially because (a) the field has developed rapidly and sparked extensive conversations 
pertaining to education and educational technology, (b) meaningful research results appear to be sparse 
(Jona & Naidu, 2014), and (c) researchers studying MOOCs reside in diverse disciplines (Veletsianos & 
Shepherdson, 2015). 

The goal of this paper is to address a number of gaps in the scholarly understanding of MOOCs and 
present a comprehensive picture of the literature by examining the geographic distribution, publication 
outlets, citations, data collection and analysis methods, and research strands of empirical research 
focusing on MOOCs. We tackle this goal by reviewing relevant literature and situating this study in the 
context of prior literature; presenting our research questions and describing the methods used to collect 
data; describing the data analysis methods used to answer each research question; and presenting our 
results. We conclude by discussing findings and making recommendations for researchers studying 
MOOC-related topics. 


Review of Relevant Literature 

We divide past literature in two sections. The first section examines and summarizes prior previous 
syntheses of the MOOC literature. The second section identifies specific gaps in the literature that are 
addressed by this paper. 

Past Systematic Analyses of the MOOC Literature 

A number of other researchers have attempted to analyze the MOOC literature, most notably, Ebben and 
Murphy (2014), Hew and Cheung (2014), Jacoby (2014), Kennedy (2014), and Liyanagunawardena, 
Adams, and Williams (2013). These reviews have focused on diverse aspects of the literature. For 
example, Hew and Cheung (2014) examined students’ and instructors’ perspectives, while Jacoby (2014) 
focused on the evidence for the role of MOOCs as a disruptive force. Despite this diversity between 
individual reports, a number of themes emerge across some or all of these reviews. We summarize the 
most salient of these themes: distinctions between cMOOCs and xMOOCs, impacts of MOOCs on 
education, demographics of MOOC users, and challenges for MOOCs. 

Distinctions between cMOOCs and xMOOCs. Each of the five previous reviews of the 
MOOC literature we identified made note of the distinction between two strands of MOOCs: cMOOCs and 
xMOOCs. cMOOCs are described as being “based on principles of connectivism, openness, and 
participatory teaching” (Jacoby, 2014, p. 76), and “[emphasizing] human agency, user participation, and 
creativity through a dynamic network of connections afforded by online technology” (Ebben & Murphy, 


199 



2014, P- 333 )- By contrast, xMOOCs are described as “follow[ing] a cognitivist-behaviorist approach” 
(Hew & Cheung, 2014, p. 50) and resemble “traditional teacher-directed course[s], yet automated, 
massive, and online” (Kennedy, 2014, p. 8). Early MOOCs tended to follow the cMOOC model, whereas 
more recently the number of xMOOCs delivered has been growing rapidly. This chronological 
categorization of cMOOCs and xMOOCs led Ebben and Murphy (2014) to describe them as two distinct 
phases of the MOOC phenomenon, while Kennedy (2014) noted that most of the nascent research on 
MOOCs had necessarily focused on the cMOOC variety. Most of the reviews focus on the philosophical 
(e.g., different approaches to openness; see next theme) and practical (e.g., different uses of technology; 
different forms of assessment) differences in the creation and delivery of cMOOCs and xMOOCs. 

One distinction between cMOOCs and xMOOCs that was discussed in a number of the reviews was the 
concept of openness. Jacoby (2014) and Kennedy (2014) both describe openness as a core component of 
cMOOCs, and something that is less present (or at least, defined differently) in xMOOCs. Jacoby (2014) 
cites literature where openness is defined with regard to transparency, course delivery, access to courses, 
course content, the manner of instruction, the way assessment is conducted, and how success is defined 
(i.e., whether there are objective or subjective criteria for succeeding in a MOOC). Ebben and Murphy’s 
(2014) review identifies some slightly different characteristics of openness, including open licenses for 
course materials, open access to courses, and the “locus and practice of knowledge acquisition and 
production” (p. 337). These authors suggest that the last of these is a hallmark of cMOOC philosophy, 
whereas the first two are common to both cMOOCs and xMOOCs. 

However, the review literature also indicates that there are common elements cMOOCs and xMOOCs 
share. For instance, instructors in both cMOOCs and xMOOCs tend to provide course outlines describing 
the general structure of the course. Where they differ in this respect is in the fact that the content to fill 
this structure is generally provided by the instructor for xMOOCs, but by the students for cMOOCs (Hew 
& Cheung, 2014). Liyanagunawardena et al. (2013) found that the literature published in 2008-2012 did 
not clearly define the different kinds of MOOCs, and Ross, Sinclair, Knox, and Macleod (2014) note that 
the differences between cMOOCs and xMOOCs are unclear. For the purposes of this paper, we will view 
MOOCs as an evolving ecosystem of online learning environments featuring open enrollment, 
characterized by a spectrum of course designs ranging from networks of distributed online resources 
(cMOOCs) to structured learning pathways centralized on digital platforms (xMOOCs). 

Impacts of MOOCs on education. A further issue discussed in many of the reviews is the 
potential impact of MOOCs on education more broadly. Jacoby (2014) specifically focuses on the 
disruptive potential of MOOCs, and so a large portion of her review is relevant to this issue. For instance, 
she identifies characteristics of some MOOCs such as their size, automation in grading, and their 
openness (particularly with regard to cMOOCs) as factors with the potential to affect approaches to 
teaching and learning. The size and openness of MOOCs are also highlighted by Kennedy (2014) in terms 
of their potential to “[disrupt] conventional thinking about the role, value, and cost of higher education” 
(p. 9). Ebben and Murphy (2014) discuss semantic shifts in the discourse around MOOCs (e.g., referring 
to students as participants ) and suggest these could imply a diminution of the authority and importance 
of the educational leader (now instructor or facilitator rather than professor ). On an institutional level, 
Jacoby describes impacts the spread of MOOCs may have on the business models of universities. She 



discusses the potential for new entrants to the higher education market to provide a product that is a 
suitable substitute for existing models of educational delivery, but also suggests that the collaboration of 
traditional institutions in creating and disseminating MOOCs may undermine this substitution. 

Demographics of MOOC users. Early research on MOOCs was in the form of institutional 
reports, and these frequently reported learner enrollments and demographics (Gasevic, Kovanovic, 
Joksimovic, & Siemens, 2014). Some of the literature reviews identified presented information about the 
demographic characteristics of MOOC users as well. Ebben and Murphy (2014) describe people from 194 
countries being enrolled in one MOOC, “[t]he vast majority [of whom] were male, between the ages of 20 
and 40, and had already earned a college degree or higher” (p. 338). Ebben and Murphy review other 
research indicating that more than half of learners in MOOCs are from countries other than the United 
States. As a counterpoint to this, Liyanagunawardena et al. (2013, p. 217) state that in the research that 
has presented demographic information, “a large majority of participants were from North America and 
Europe,” with a small minority being from Asia, South East Asia, or Africa. These authors suggest that the 
reasons for this may be technological and linguistic. 

Challenges for MOOCs. All of the reviews identify challenges or potential challenges for 
MOOCs to overcome. One of the most salient of these relates to course completion. Ebben and Murphy 
(2014) review research suggesting that completion rates in MOOCs are less than 10%. They suggest that 
this may result from the fact that participation in MOOCs is free, leading users to participate in activities 
that are of interest to them without necessarily completing all parts required to complete a course. 
However, Hew and Cheung (2014) are less sanguine about participants’ reasons for non-completion, 
identifying the following list of reasons as relevant: “a lack of incentive, insufficient prior knowledge (e.g., 
lack of math skills), a lack of focus on the discussion forum (e.g., off-track posts), failure to understand the 
content and [having] no one to turn to for help, ambiguous assignments and course expectations, and a 
lack of time due to having other priorities and commitments to fulfil” (p. 49). Other than course non¬ 
completion, other challenges mentioned in the reviews include economic challenges (e.g., the high cost of 
running a MOOC, or lack of a business model; Hew & Cheung, 2014; Jacoby, 2014), limitations of mass 
teaching methods (Kennedy, 2014), accreditation (Liyanagunawardena et al., 2013), and the assessment 
of complex writing such as essays (Ebben & Murphy, 2014). 

Gaps in the Literature 

While prior reviews of the literature provide a useful survey of the field, none has focused exclusively on 
empirical literature in MOOCs. This paper represents the first effort to review the empirical literature on 
MOOCs for a particular time period to understand various structural components of it, and as such fills a 
gap in the literature overall. While early writing on MOOCs was primarily conceptual (Kennedy, 2014), 
this is no longer the case, and the field will benefit from an early review of the status of the evidence-based 
literature. Furthermore, this research addresses five specific gaps that we identified in our examination of 
the current scholarly literature. These gaps can be resolved by a systematic review of the literature, and 
they are described next. 

Geographic distribution. MOOCs have often been proposed as democratizing vehicles 
intended to provide free or inexpensive education. Liyanagunawardena et al. (2013), however, indicate 
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that research arising predominantly from Western authors will largely serve countries where learners 
have access to digital technologies, understand the language, and identify with a Western learning culture. 
If authors from select geographic regions dominate the direction and focus of the MOOC research, the 
development of empirical understanding of MOOCs might be limited in scope (cf. Zawacki-Richter, 
Backer, & Voigt, 2009). Contributions to MOOC research from a broad geographic area may provide a 
more diverse perspective through which to make sense of the MOOC phenomenon. Nevertheless, only a 
few articles have examined the geographic distribution of MOOC-related efforts and these investigations 
occurred in particular contexts and only for the literature published up to 2012. In analyzing grant 
applications submitted to the MOOC Research Initiative for example, Gasevic et al. (2014) found that the 
geographic distribution of authors was heavily concentrated in North America, Europe, and Asia with 96% 
of accepted proposals originating from those three regions. These results are similar to findings pertaining 
to distance education in general. For example, in a review of the research published in distance education 
journals in 2000-2008, Zawacki-Richter, Backer , and Vogt (2009) found that more than 80% of the 
articles are authored by individuals from five countries (USA, Canada, UK, Australia, and China). We 
were unable to identify further research examining the geographic location of MOOC authors. 

Publication outlets. Even though the three ways that all disciplines predominantly use to 
disseminate research findings appear to be via journal articles, book chapters, and conferences (Sparks & 
House, 2005), the choice of publication outlets varies significantly across disciplines (Kling & McKim, 
2000). Because the MOOC phenomenon has gained interest from academics in numerous disciplines, we 
were interested in examining which communicative forums were used by researchers conducting MOOC 
research. In examining the 2008-2012 literature Liyanagunawardena et al. (2013) reported that the 
majority of outlets published only one paper on MOOCs, while more were published by the International 
Review of Research in Open and Distrubuted Learning 1 (six), the European Journal of Open Distance 
and E-Learning (three), and the International Conference on Networked Learning (three). These fora 
focus on online education and educational technology, and MOOC research has gained interest from more 
disciplines since 2012 (Veletsianos & Shepherdson, 2015). An examination of the communicative forums 
of MOOC research enables us to examine in which outlets MOOC research has been published and 
examine whether the early findings reported by Liyanagunawardena et al. still hold true. Because 
conference proceedings have a quicker turn around time and may have greater impact in an emerging 
research area (Ssebo, Rose, & Flak, 2008), as well as due to the fact that some journals and conferences 
published special issues that focused on MOOCs, we expect to see (a) conferences proceedings, and (b) 
journals that have published special issues on MOOCs, featuring prominently. 

Citations. Scholars use various tools to estimate the reach and impact of scholarship. Paper 
citation counts are frequently used for this purpose, and even though they are imperfect metrics of impact 
(e.g., they do not distinguish between positive and negative citations [Smeyers & Burbules, 2011; Togia & 
Tsigilis, 2006]), they are helpful as a way to begin examining the literature in the field, and identifying 
papers that, for one reason or another, are popular. In analyzing the distance education literature, 
Bozkurt et al. (2015) also argued that citation analyses may provide a reference guide and reading list to 

1 This journal used to be called International Review of Research in Open and Distance Learning and was renamed 
in 2015. 



examine the field. Gasevic et al. (2014) examined the papers cited most frequently in the MOOC Research 
Initiative submissions and suggested that the most cited papers appear to be those that were most 
relevant to the call for awards. This investigation fills a gap in the scholarly literature by presenting the 
most highly cited papers in the field at the time of writing. 

Data collection and analysis methods. The data collection and data analysis methods of 
MOOC research are areas that are poorly investigated in prior literature. The one paper (Gasevic et al., 
2014) that examined research methods in the area categorized MOOC research proposals submitted for 
funding as using mixed (42.3%), quantitative (33.3%), or qualitative (24.4%) methods. While such a 
categorization provides a useful picture of the research in the field, it does not distinguish between data 
collection and data analysis methods. This is problematic because the type of data collected does not 
necessarily reflect the analytic methods used. For instance, qualitative data may be converted to 
quantitative data (e.g., counting the number of times a participant expresses a negative feeling about a 
course). Distinguishing between data collection and data analysis methods will add nuance to our 
understanding of how researchers have come to know what they know about MOOCs. While Bates (2014) 
highlighted the diversity of research methodologies that existed in a special issue on MOOCs, Veletsianos, 
Collier, and Schneider (2015, p. 574) argued that “ease of access to large data sets from xMOOCs offered 
through an increasing number of centralized platforms has shifted the focus of MOOC research primarily 
to data science and computational methodologies” and claimed that “the MOOC phenomenon 
experienced a surge of research using quantitative, clickstream and observational data.” By investigating 
data collection and data analysis methods, this research will be able to empirically validate this claim. 

Research strands. Liyanagunawardena et al. (2013) and Gasevic et al. (2014) examined the 
focus of MOOC-related literature, though those reviews did not focus exclusively on empirical research. 
By identifying the major research strands in the empirical literature, this paper will enable researchers to 
ascertain the areas that have attracted the (a) greatest attention, and (b) least attention in the field. By 
identifying these two areas, we can describe the interests of researchers, the areas that may need greater 
attention, and the areas in which scholarly understanding of MOOCs is grounded on an evidence base. 

Liyanagunawardena and colleagues categorized the papers they identified into eight categories: 
Introductory, Concept, Case studies, Educational theory, Technology, Participant-focused, Provider- 
focused, and Other. The majority of papers fell into Case studies and Educational theory (a category 
focusing on pedagogical approaches used in MOOCs). Gasevic and colleagues identified five strands in 
MOOC proposals submitted for grant funding: engagement and learning success, MOOC design and 
curriculum, self-regulated learning and social learning, Social Network Analysis and networked learning, 
and motivation, attitude and success criteria. 


Research Questions 

To address these gaps in the literature, we pose the following research questions (RQ), each 
corresponding to a particular gap: 
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RQi: How is empirical MOOC research geographically distributed? 


• RQ2: Is empirical MOOC research usually published in journal or conference proceedings? In 
which journals and conference proceedings is MOOC research currently being published? 

• RQ3: Which empirical MOOC studies are cited the most? 

• RQ4: What data collection methods and data analysis methods are used in empirical studies of 
MOOCs? 

• RQ5: What are the research strands of empirical MOOC research? 


Methods 

In this section we describe the approaches we took to answer the research questions. We describe the 
systematic methods we used to gather literature (data collection) and the analytic methods we used to 
examine the literature corpus we gathered (data classification and analysis). 

Data Collection 

Literature discovery searches were conducted using the key words “MOOC” or “Massive Open Online 
Course.” To be included in the corpus, each identified document ought to focus on MOOCs, and ought to 
have been (1) empirical, (2) published in a peer-reviewed journal, in conference proceedings, or in 
Educause Review 2 , (3) published or was available online as in press between January 2013 and January 
2015, and (4) written in English. We defined empirical papers as those that gathered and analyzed 
primary or secondaiy data in their investigation. Using this definition, conceptual and theoretical papers 
did not meet the inclusion criteria. The majority of the papers that we discovered in the literature search 
were not empirical (and were thus excluded). 

Three trained researchers engaged in the literature discovery process. As each individual encountered a 
paper, she or he examined its abstract to determine whether it fit the inclusion criteria. If a determination 
could be made by examining the abstract, the document was added to a shared computer folder. If no 
determination could be made by examining the abstract, the paper was downloaded and the full paper 
was examined. All identified papers were examined by two researchers to ensure consensus that they fit 
the inclusion criteria. 

The Scopus database was searched first. Out of 433 results returned, 81 were deemed to fit the inclusion 
criteria. In their review of the 2008-2012 MOOC literature Liyanagunawardena et al. (2013) searched the 
following four journals specific to educational technology for MOOC-related research: The American 


2 Even though Educause Review is a professional magazine (i.e. not a journal and neither a conference), it was included 

here because it was observed that it published empirical papers on MOOCs. While the quality of professional magazines may 
vary widely, Educause Review is regarded highly by educational technology scholars (Perkins & Lowenthal. in press). 



Journal of Distance Education, the British Journal of Educational Technology, Distance Education, and 
the Journal of Online Learning and Teaching. The first three journals are indexed by Scopus and thus 
any papers that fit the inclusion criteria would have already been identified. The Journal of Online 
Learning and Teaching was not indexed by Scopus at the time of writing. We also examined this 
journal using the search criteria described above and identified seven papers that fit the inclusion 
criteria. These were added to our corpus. 

Next, we used the Summon search engine. Summon is a one-stop gateway to institutional resources. 
While it is difficult to replicate Summon searches (each institution will likely have a different collection of 
databases feeding into its discovery layer), we were interested in collecting all the possible available 
literature on MOOCs, and saw Summon as yet another way to discover such literature. Summon 
generated 1337 results and out of those, we exported 505 results for further analysis. Of those, 10 new 
papers fit our inclusion criteria and were added to the corpus. 

We followed that search with a Google Scholar search using the same keywords. Unlike Scopus, Google 
Scholar does not provide a definitive list of what exactly it indexes, but it still provides a source for 
locating grey literature that may have been difficult to locate through commercial publisher databases 
(e.g., Scopus), as well as literature beyond the scope of any one library’s collection. We were able to locate 
11 additional papers via this method. We ended our search at the 200th record as results were becoming 
increasingly irrelevant or redundant beyond that point. 

Next, we searched two stand-alone libraries (EdITLib Digital Library and the Educause Library), both of 
which focus on educational technology materials. The EdITLib Digital Library provides access to an 
extensive library of conference proceedings. We were able to identify five relevant papers from the 
EdITLib Digital Library and 4 papers from the Educause Library that fit the aforementioned inclusion 
criteria. 

Next, we engaged in a forward referencing process, to identify relevant papers that cited the papers that 
we had already located. This process was used by Gao, Luo, and Zhang (2012) in their examination of the 
microblogging literature and by Liyanagunawardena et al. (2013) and worked as follows: We located each 
one of the papers we already identified (original) in Google Scholar. This service provides information on 
how many times a paper is cited (Figure 1) and allows researchers to view all papers citing the original. 
We identified the 120 papers that were in our corpus at the time, located one by one in Google Scholar, 
and examined all the papers that cited each original. If the paper fit our inclusion criteria, we included it 
in our database. This process returned 60 additional documents. We used Google Scholar instead of 
Scopus for our forward referencing check because the former appears to include more grey literature and 
provide greater coverage than the latter. 

Where is research on massive open online courses headed? A data analysis of the MOOC 
Research Initiative 

D Gasevic . V Kovanovic . S Joksimovic ... - The International Review .... 2014 - irrodl.org 
Abstract This paper reports on the results of an analysis of the research proposals submitted 
to the MOOC Research Initiative (MRI) funded by the Gates Foundation and administered by 
Athabasca University. The goal of MRI was to mobilize researchers to engage into critical... 

| Cited by 15~| Related articles All 8 versions Cite Save More 
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Figure 1 . An example of how Google Scholar was used in a forward referencing process 


Finally, we conducted a completeness search (as opposed to a discovery search): We examined the 
references of the 17 papers published in 2015 that were in our corpus to identify any papers that we may 
have missed. Since these papers were either already published in 2015 or in press at the time of writing, 
they were more likely to reference literature published in 2013-2014 than the papers published in 2013 or 
2014. Five new papers were identified that fit the inclusion criteria and these were added to the corpus. 

The final number of published papers that constituted the corpus of this study was 183. Thirty-seven were 
published in 2013,129 were published in 2014, and 17 were in press at the time of data collection. Because 
we synthesized the results of all literature analyses published at the time in our literature review above, we 
have excluded them from our corpus. The literature search process, numbers of papers identified, and 
date on which the data collection process was completed are summarized in Table 1. As a point of 
comparison, Liyanagunawardena et al. (2013) conducted a nearly identical data collection process for all 
the MOOC literature that they could identify for 2008-2012 and were able to locate only 45 articles at the 
time. These articles were located in journals (17), conference proceedings (13), academic magazines (10), 
reports (3), and conference workshops (2). 

Table 1 

Data Collection Methods, Results, and Dates 


Method 

# Identified 

Date 

Search: Scopus 

81 

7 January 2015 

Search: Journal of Online Learning and 
Teaching 

7 

11 January 2015 

Search: Summon 

10 

11 January 2015 

Search: Google Scholar 

11 

11 January 2015 

Search: EdITLib Digital Library 

5 

11 January 2015 

Search: Educause Library 

4 

13 January 2015 

Forward Referencing search 

60 

15-18 January 2015 

Reference list check 

5 

1 February, 2015 


Data Classification and Analysis 

The 183 papers collected were classified and analyzed in both quantitative and qualitative ways. We 
describe each analytic method employed in relation to each particular research question posed. 

To determine the geographical distribution of MOOC research (RQi), we coded the affiliations of authors 
from our corpus (n = 460) in two ways: by the countiy in which their institution or organisation was 
located (or, if unaffiliated, by the countiy in which the author was located), and by the associated region. 
To determine the publication outlets of MOOC research (RQ2), we classified each publication according to 
whether it was published in a journal or conference proceedings, and counted the times each outlet 



appeared in our corpus. To determine which studies were cited the most (RQ3), we identified each paper 
in our corpus on Google Scholar and noted its citation count (Figure 1). 

To determine the data collection methods used in the identified corpus (RQ4), two researchers coded the 
corpus using an 8-item coding scheme. Tashakkori and Teddlie (2003) identified six data collection 
methods, and that was used as a basis of creating a coding scheme. Tashakkori and Teddlie identified the 
following methods: questionnaires and surveys, interviews, focus groups, tests, and observations. These 
authors also included secondary data as a data collection method, but based on our understanding of the 
literature and our understanding that trace data are increasingly used in digital contexts, we differentiated 
between the automated collection of secondaiy data (e.g., trace data collected by digital platforms) and 
human collection of secondaiy data (e.g., use of photographs). This raised our number of data collection 
methods to seven. To capture data collection methods that did not fit into any of the above categories, we 
used an eighth code named Other. Using these codes, independently, each researcher examined each 
paper and classified it according to its data collection methods. Papers were assigned between one and 
five data collection method codes. Researchers assigned the codes a total of 642 times. Inter-rater 
agreement was calculated to be 68.5%. Next, the two coders came together to reconcile differences. Each 
difference was discussed and resolved, until both coders were satisfied that the codes assigned 
appropriately described the data collection methods used in each manuscript. The final dataset consisted 
of the eight codes assigned 324 times. 

To determine the data analysis methods used in the identified corpus (RQ4), two researchers used a 
coding scheme consisting of 11 categories. These categories were compiled by the researchers after 
consulting methodological resources (e.g., Merriam, 2002) and prior MOOC research (e.g., Gasevic et al., 
2014). Each category described a data analysis method. The categories were: Basic qualitative study, 
grounded theory, phenomenology, ethnography, discourse analysis, experimental and quasi- 
experimental, correlational, natural language processing, social network analysis, descriptive statistics, 
and other. Again, each researcher examined each paper independently and classified it according to its 
data analysis methods. The two researchers assigned the 11 categories a total of 830 times. Interrater 
agreement was calculated to be 78.7%. The two researchers discussed and reconciled differences until 
consensus was reached on all codes. The final dataset consisted of the 11 codes assigned 439 times. 

To determine the research strands in the identified corpus (RQ5), two researchers independently read and 
assigned emerging codes to each paper. The codes described the focus of each paper, and there were no 
pre-determined limits set on the number of codes to be assigned to each paper. The first researcher 
generated 25 codes and the second researcher generated 31 codes. Next, they met to discuss their findings 
and identify categories describing their codes. They identified five categories describing strands of extant 
MOOC research: student-focused, teacher-focused, design-focused, context and impact, and other. Next, 
they returned to the papers and independently assigned papers to each theme. Inter-rater agreement was 
calculated to be 77.9%. Next, they discussed discrepancies and resolved them, reaching agreement on all 
papers. The final dataset consisted of the five codes assigned for a total of 291 times. 

Limitations 
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This study clarifies the state of the literature published at a particular point in time using a particular 
methodology. There are four limitations arising from the research context. First, this study draws upon 
less than three years of data and its findings are only representative of the research on MOOCs during this 
period—the literature will likely change over time in the same way that it has changed since 
Liyanagunawardena et al. (2013), published their own analysis. This limitation is inevitable as new work 
in the area is quickly emerging. Second, the data analysis methods used in this study do not allow us to 
judge the quality of the research reported. It should be recognized therefore, that the papers included in 
our corpus are of mixed quality. For instance, our reporting on the use of grounded theory does not 
necessarily examine whether the authors used the method correctly, rigorously, or even uniformly. Third, 
while our data reflect some of the content of the papers analyzed (e.g., what data collection methods were 
used), they do not reflect a full evaluation of the contents of the papers (e.g., what were the results 
reported in each research strand identified). Fourth, while non-English native speakers author MOOC 
papers in English, the choice to exclude papers written in languages other than English may have limited 
the size and diversity of the sample. 


Results 

RQl: How is Empirical MOOC Research Geographically Distributed? 

This question was asked to identify whether the empirical research on MOOCs centered in one countiy or 
region, or whether it was a global phenomenon. The vast majority of authors came from North America 
and Europe, which between them accounted for over 82% of the author affiliations (Table 2). 

Table 2 

Frequency (Percentage) of Each Region among Author Affiliations^ 


Region 

Corpus 

North America 

254 ( 55 - 2 %) 

Europe 

124 (27.0%) 

Asia 

37 (8.0%) 

Oceania 

35-5 ( 7 - 7 %) 

Middle-east 

4 (0.8%) 

Africa 

3 (0.6%) 

South America 

2.5 (0.5%) 


Table 3 shows that more than half of the authors were affiliated with institutions from the USA and 
between 1% and 10% of the authors came from each of the United Kingdom (10%), Australia (7.7%), China 
( 5 - 4 %), Spain (4.8%) Canada (4.5%), Germany (2.2%), Switzerland (1.3%), and the Netherlands (1.1%). 


3 Decimal values reflect authors with affiliations in multiple regions (e.g., 0.5 added to each region’s count for an author 

with affiliations in two separate regions). 



These nine countries represented 87.2% of the author affiliations. The other 12.8% of authors represented 
29 other countries that had four or fewer authors each, with each countiy having less than 1% of the 
corpus. 

Table 3 

Frequency (Percentage) of Each Country among Author Affiliations 4 


Country 

Corpus 

USA 

231 (50.2%) 

United Kingdom 

46 (10%) 

Australia 

35-5 ( 7 - 7 %) 

China 

25 ( 5 - 4 %) 

Spain 

22 (4.8%) 

Canada 

21 (4.5%) 

Germany 

10 (2.2%) 

Switzerland 

6 ( 1 . 396 ) 

Netherlands 

5 (1.1%) 

Other (29 countries) 

58.5 (12.8%) 


RQ2: Is Empirical MOOC Research Usually Published in Journals or Conference 
Proceedings? In Which Journals and Conference Proceedings is MOOC 
Research Currently Being Published? 

Ninety-eight papers were published in peer-reviewed journals and 85 were published in conference 
proceedings. Eighty publication outlets published one item each, 16 outlets included two items each, and 
four outlets included three items each. The rest of the outlets included 4 or more items and these are 
shown in Table 4. Three of these outlets are peer-reviewed journals that focus on online or distance 
education and published special issues on MOOCs during the period under investigation. Four of these 
outlets are conferences with one focusing specifically on learning at scale and MOOCs (L@S ‘14). 

Table 4 

Outlets Publishing MOOC Research 


Outlet name 

Number of 
Papers 

Type 

International Review of Research in Open and Distributed 
Learning (IRRODL) 

18 

Journal 

Proceedings of the first ACM conference on Learning @ scale 
(L@S ’14) 

11 

Conference 

Journal of Online Learning and Teaching (JOLT) 

7 

Journal 

Conference on Empirical Methods in Natural Language 

5 

Conference 


4 Decimal values reflect authors with affiliations in multiple countries (e.g., 0.5 added to each country’s count for an 

author with affiliations in two separate countries). 
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Processing (EMNLP) 

Distance Education 5 

NIPS Workshop on Data Driven Education 5 

Proceedings of the 2014 ASCILITE Conference 4 

EDUCAUSE Review_4 


Journal 

Conference 

Conference 

Journal 


RQ3: Which Empirical MOOC Studies Are Cited the Most? 

At the time of writing, of the 183 papers identified, eighty-seven (47.5%) were cited zero times. Seventy- 
two papers were cited one to ten times, with the majority being cited once (16), twice (12), or thrice (14). 
Ten papers were cited 11 to 20 times. The rest of the papers (13) were cited 25 or more times. These are 
shown in Table 5. For comparative purposes, we are including a column that shows the number of times 
these papers were one year after the data were collected and right before this paper was published. Two of 
these papers were published in 2014 and the rest were published in 2013. Seven of the thirteen papers 
were published in conference or workshop proceedings and six were published in journals. While this 
analysis focuses on a different time period than that of Gasevic et al. (2014; i.e., the grant applications 
examined by Gasevic et al. were for grants awarded in late 2013, and therefore papers cited would have 
been published before then) and we limit our corpus to empirical papers, the two most cited papers 
identified here (Breslow et al., 2013; Kizilcec, Piech, & Schneider, 2013) were also highly cited in Gasevic 
et al.’s corpus. 

Table 5 

MOOC Publications Cited Most Frequently 


Paper 


Number of 
citations 

(Feb., 2015) 


Number of 
citations 
(Feb., 2016) 


Breslow, L., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A. D., 124 

& Seaton, D. T. (2013). Studying learning in the worldwide 
classroom research into edX’s first MOOC. Research & Practice in 
Assessment, 8(1), 13-25. 

Kizilcec, R. F., Piech, C., & Schneider, E. (2013). Deconstructing 117 

disengagement: Analyzing learner subpopulations in massive 
open online courses. In 3rd International Conference on 
Learning Analytics and Knowledge, LAK 2013 (pp. 170-179). 

New York, NY: Association of Computing Machinery. 

Clow, D. (2013). MOOCs and the funnel of participation. In 3rd 83 

International Conference on Learning Analytics and Knowledge: 

LAK 2013 (pp. 185-189). New York, NY: Association of 
Computing Machinery. 

73 

Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., & Roller, D. (2013). 

Tuned models of peer assessment in MOOCs. In Proceedings of 
The 6th International Conference on Educational Data Mining 
(pp. 2579-2585). Dallas, TX: IEEE. 


338 


265 


195 


154 



97 


Singh, R., Gulwani, S., & Solar-Lezama, A. (2013). Automated 
feedback generation for introductory programming assignments. 
In Proceedings of the 34th ACM SIGPLAN Conference on 
Programming Language Design and Implementation (pp. 15— 
26). New York, NY: Association of Computing Machinery. 

Roller, D., Ng, A., Do, C., & Chen, Z. (2013). Retention and 
intention in Massive Open Online Courses: In depth. EDUCAUSE 
Review. Retrieved from 

http://er.educause. edu/articles/2013/6/retention-and-intention- 
in-massive-open-online-courses-in-depth 

Yang, D., Sinha, T., Adamson, D., & Rose, C. P. (2013). “Turn on, 
tune in, drop out”: Anticipating student dropouts in Massive 
Open Online Courses. In NIPS Workshop on Data Driven 
Education (pp. 1-8). Lake Tahoe, NV: Neural Information 
Processing Systems Foundation. 

Bruff, D. O., Fisher, D. H., Mcewen, K. E., & Smith, B. E. (2013). 
Wrapping a MOOC: Student perceptions of an experiment in 
blended learning. Journal of Online Learning and Teaching, 

9(2), 187-199- 

Griinewald, F., Meinel, C., Totschnig, M., & Willems, C. (2013). 
Designing MOOCs for the support of multiple learning styles. In 
8th European Conference on Technology Enhanced Learning, 
EC-TEL 2013 (pp. 371-382). Paphos, Cyprus: European 
Association of Technology Enhanced Learning. 

Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., 

Cheng, J., Roller, D., & Rlemmer, S. R. (2013). Peer and self 
assessment in massive online classes. ACM Transactions on 
Computer-Human Interaction (TOCHI), 20(6), 1—31. 

Milligan, C., Littlejohn, A., & Margaryan, A. (2013). Patterns of 
engagement in connectivist MOOCs. Journal of Online Learning 
and Teaching, 9(2), 149-159. 


56 


41 


106 


35 109 


34 84 


29 59 


29 98 


29 


91 


Guo, P. J., & Reinecke, R. (2014). Demographic differences in 27 47 

how students navigate through MOOCs. In Proceedings of the 
first ACM conference on Learning @ scale conference—L@S ’14 
(pp. 21-30). New York, NY: ACM Press. 

DeBoer, J., Ho, A. D., Stump, G. S., & Breslow, L. (2014). 25 78 

Changing “course”: Reconceptualizing educational variables for 
massive open online courses. Educational Researcher, 43(2), 74- 
84 - 


RQ4: What Data Collection Methods and Data Analysis Methods Are Used in 
Empirical Studies of MOOCs? 

Data collection methods. The majority of the papers used one (44.8%) or two (38.3%) data 
collection methods. The rest used three (13.1%), four (2.7%), and five (1.1%) data collection methods 
respectively. Automated collection of secondaiy data (e.g., trace data) was used most frequently (73.2% of 
papers used this method). The second most popular data collection method was questionnaires/surveys, 
which were used in 55.7% of papers. The rest of the data collection methods were used much less 
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frequently: Interviews (13.7%), secondaiy data collected by humans (13.7%), tests (8.7%), observations 
(4.9%), focus groups (4.4%), and other (2.7%). 

Automated methods of data collection were used as the sole data collection method in 26.8% of the 
corpus. They were also used in combination with one (31.7%), two (11.5%), three (2.1%), and four (1%) 
other methods. Questionnaires and surveys were used as the sole data collection method less frequently 
than automated methods (10.4%). They were used in conjunction with one (28.9%), two (12.5%), three 
(2.7%), and four (1%) other methods. 

Data analysis methods. The majority of the papers used two (47%) or three (28.4%) data 
analysis methods. The rest used one (13.1%), four (8.7%), and five (2.7%) analytic methods. Table 6 shows 
the data analysis methods used in this corpus ranked in order of frequency. Descriptive statistics were 
reported in almost all papers, though follow up analysis showed that they were used as the sole method of 
analysis in only 7.7% of papers. Correlational, basic qualitative, and experimental or quasi-experimental 
methods were also used frequently. 

Table 6 

Frequency (Percentage) of Data Analysis Methods Used 


Analytic Method 

Frequency (%) of 
Total Papers 

Descriptive statistics 

93-4 

Correlational 

52.5 

Basic qualitative study 

38.8 

Experimental and quasi-experimental 

25-7 

Grounded Theory 

7.6 

Natural Language processing 

7.6 

Social Network Analysis 

6.6 

Ethnography 

4-4 

Phenomenology 

2.2 

Discourse analysis 

1.0 


RQ5: What are the research strands of empirical MOOC research? 

We identified five categories describing the research reported in the corpus: student-focused; design- 
focused; context and impact; instructor-focused; and other (Table 7). 

Table 7 

Research Strands Present in the Empirical MOOC Literature and Associated Frequency (Percentage) of 
Each Strand (as a Percentage of Total) 


Research Strand 


Student-focused 
Design-focused 
Context and impact 
Instructor-focused 


Frequency (%) of 
Total Papers 

83.6 

46.4 

10.9 

8.2 



Other 


9-8 


Student-focused. The majority of the literature focused on topics related to learners and 
learning. Student-related areas were addressed in 83.6% of the papers. Topics covered in this theme 
included research on learner behaviors, performance, learner participation and interaction, learner 
perceptions and preferences, learner experiences, motivation, and demographics. Two popular areas of 
research under these themes were the topics of (a) completion and retention, and (b) learner 
subpopulations. 

Given the attention that MOOC completion rates have received in the mass media, it is perhaps 
unsurprising that numerous researchers have examined completion, retention, and learner 
subpopulations. For instance, Jordan (2014) and Perna et al. (2014) found low completion rates (often 
less than 10% of registrants), and Reich (2014) showed that certification rates vary substantially among 
learners with different intentions. This line of research is often related to researchers’ attempts to identify 
and classify learners into various groupings. Kizilcec, Piech, and Schneider (2013) were the first to 
examine this topic and classified learners in four groups: completing, auditing, disengaging, and 
sampling. Other authors identified other engagement and participation patterns. For instance, Alario- 
Hoyos, Perez-Sanagustin, Delgado-Kloos, Parada, and Munoz-Organero (2014) classify learners as no- 
shows, observers, drop-ins, latecomers, drop-in latecomers, non-engaged, and engaged, while Kellogg, 
Booth, and Oliver (2014) classified educators who participated in two education-focused MOOCs as 
reciprocators, networkers, broadcasters, and invisible. 

Design-focused. Nearly half (46.4%) of the papers identified in the literature search had some 
focus on topics relating to the design, creation, and implementation of MOOCs themselves, which we 
collapsed into a broad design-focused theme. This included research pertaining to methods of assessment, 
the description of unique learning environments, the creation of MOOCs on specific topics, and the 
evaluation of course success. It also comprised research on the use of MOOCs in blended learning, and 
reports of novel technological aids to teaching in MOOCs. 

Common within this theme was research that investigated the utility of individual elements of MOOCs. 
For example, some researchers focused on the inclusion of tools for social interaction within online 
courses (e.g., Yang, Piergallini, Howley, & Rose, 2014), while others investigated the use of specific types 
of media in instruction (e.g., Guo Kim, & Rubin, 2014; Kim, Guo, Seaton, Mitros, & Gajos, 2014). There 
was also a strong focus on means of assessing student work (e.g., Admiraal, Huisman, & Van de Ven, 
2014; Cisel, Bachelet, & Bruillard, 2013). 

Context and impact. Content pertaining to the context and impact (both social and 
educational) of MOOCs occurred in 10.9% of the corpus. This included research into perceptions of 
MOOCs (e.g., Bulfin, Pangrazio, & Selwyn, 2014; Radford et al., 2014), their usefulness as an educational 
medium (e.g., Subhi et al., 2014; Wang, Paquette, & Baker, 2014), and their economic impact (e.g., 
Hollands & Tirthali, 2014). 
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Instructor-focused. Very few papers focused on topics related to instructors and teaching 
(8.2%). Papers within this theme largely focused on academics’ awareness, perspectives of, and 
experiences with MOOCs (e.g., Khalil & Ebner, 2013; Sheard et al., 2014; Stanton & Harkness, 2014; 
Stephens-Martinez, Hearst, & Fox, 2014). 

Other. Papers with content that could not be categorized into student-focused, design-focused, 
instructor-focused, or context and impact, were included in a final theme that we called other (9.8%). This 
theme included papers examining issues pertaining to MOOCs and institutions of higher education (e.g., 
O’Connor, 2014) and meta-research papers examining MOOC research and focus areas (e.g., Gasevic et 
al., 2014). All but two of the papers with content classified as other also contained content belonging to 
one or more of the four previously-described themes. 


Discussion 

We gathered the empirical literature on MOOCs published in 2013-2015 and identified 183 publications— 
many more than were identified in research examining prior time periods (e.g., 45 were available at the 
time Liyanagunawardena et al. (2013) conducted their study). Using this dataset, we examined the 
geographic distribution, publication outlets, citations, data collection and analysis methods, and research 
strands of the 2013-2015 empirical research focusing on MOOCs. Results demonstrated that more than 
80% of this literature is published by individuals whose home institutions are in North America and 
Europe. Such research has been published in a wide array of journals and conference proceedings, but 
some publication outlets have published more empirical research than the rest, usually as a result of 
special issues focusing on the topic. Results also showed that a select few papers are widely cited while 
nearly half of the papers are cited zero times. Finally, results showed that the literature demonstrates an 
overwhelming dependence on particular data collection or analysis methods and particular research 
strands. These results have several implications for MOOC research and the state of the field. 

Dependence on Particular Research Methods May Restrict our Understanding 
of MOOCs 

Analysis suggests that researchers have favored a quantitative, if not positivist approach to the conduct of 
MOOC research. Survey data and secondaiy data collected via automated methods dominated the 
analyses. While some interpretive research was conducted in MOOCs in this time period, it was often 
basic. Very few studies were informed by methods traditionally associated with qualitative research 
approaches (e.g., interviews, observations, and focus groups). Thus, even though results suggest that 
research on MOOCs focuses on student-related topics, learners’ voices were largely absent in the 
literature. These results provide empirical support to the claim published by Veletsianos, Collier, and 
Schneider (2015, pp. 573) that “the MOOC phenomenon experienced a surge of research using 
quantitative, clickstream and observational data” and suggest that what we know about MOOCs may be 
the result of the field’s overwhelming dependence on particular data collection and analysis methods. 
Based on these results, we suggest that an expansion of the methodological approaches used in MOOC 
research is urgently needed. Given that research into MOOCs is expected to inform learning in all 
environments and not just MOOCs (Rose et al., 2015; Singer, 2014), a broader methodological toolkit is 



imperative. Fruitful future research endeavors in this area may focus on examining how particular 
methodologies have shaped the field, whether research methods are favored by researchers from 
particular disciplines, and some conferences and journals more than others distort the dominant 
narratives in the literature. 

There Is a Paucity of Research Examining Instructor-Related Topics 

Analysis shows that there is limited research reported on instructor-related topics. This is a rich area for 
future research. Topics of interest in this area may include instructor motivations, experiences, and 
perceptions. Researchers could examine how instructors experience the design and development of these 
courses, why they choose to teach MOOCs, and how they perceive their relationship with MOOC learners 
(and whether that relationship differs from traditional student-learner relationships). Given that a 
number of MOOCs enlist the help of instructional assistants (e.g., Teaching Assistants and course 
alumni), research in this area could investigate the impact of instructional assistants on learning and 
support, as well as the experiences that instructional staff have in the delivery of the course. 

Understanding Learner Subpopulations 

Results show that a number of researchers have attempted to identify and classify learners into various 
groupings. For instance, the literature suggests that MOOC learners can be described as completing, 
auditing, disengaging, and sampling (Kizilcec, Piech, & Schneider, 2013), or no-shows, observers, drop- 
ins, latecomers, drop-in latecomers, non-engaged, and engaged (Alario-Hoyos et al., 2014). However, 
little research examines the experiences of different populations and how and why learning experiences 
differ between groups. One exception is Huang et al. (2014) who examined the quality and impact of 
discussion forum posts made by high-volume contributors. Future research into this area for instance 
could examine why some learners disengage, how the learning experience of drop-ins differs from the 
learning experience of those who are non-engaged, and what interventions may scaffold different types of 
learners. 

cMOOCs vs. xMOOCs 

While some of the papers that we identified alluded to distinctions between xMOOCs and cMOOCs, and 
some authors focused their research on a particular design, the current literature continues to reflect the 
findings of Liyanagunawardena et al. (2013): the majority of the literature does not clearly define the 
kinds of MOOCs studied. As the field continues to develop and change, the distinctions between xMOOCs 
and cMOOCs become increasingly unclear and problematic. In particular, courses include designs, 
artifacts, and philosophies that cannot be easily categorized into cMOOCs or xMOOCs. With continued 
attempts at exploring alternative MOOC designs—such as the dual-layer MOOC (Dawson et al., 2015) or 
the MOOC in which a teaching hot contributed automated instructional, procedural, and social support to 
learners (Bayne, 2015)—the cMOOC vs. xMOOC distinction belies substantive differences between 
individual courses. 

The Geography of MOOC Research 

Our geographical analysis of author affiliations showed that over half of the authors conducted their 
research in the USA, and over 80% of authors were affiliated with institutions in North America or 
Europe. By comparison, Zawacki-Richter et al. (2009) found that over 80% of first authors of papers 
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published in distance education journals hailed from five countries: the USA, Canada, the United 
Kingdom, Australia, and China. This indicates that these fields of research are similarly concentrated in a 
few geographic regions. We compared these results to output from the SCImago Country Rankings tool 
(SCImago, 2007), which summarizes various indicators, including geographic origin, of all papers 
published in Scopus. In contrast, according to SCImago, of all citable documents published in 2013 across 
all disciplines only 18.6% came from the USA, and it takes the top 20 countries (including at least seven 
countries from outside of North America and Europe) to account for 80% of academic output. As such, 
while it appears that the MOOC literature and the distance education literature may be similar in this 
regard, this is not simply a reflection of geographical contributions to academic output in general. Future 
research in this area may examine whether and how research on MOOCs differs according to its 
geographical origins, especially as prior research indicates that conceptions of education as a discipline 
differ between regions (e.g., Biesta, 2011). While it is possible that our exclusion of literature authored in 
languages other than English limited the literature from other regions than North America and Europe, it 
is also possible that the preponderance of literature from these regions discovered in the research process 
described herein may be the result of other factors (e.g., filtering algorithms) that make some literature 
less visible than other literature. Future research may also examine whether the MOOC literature is 
biased in particular ways in favor of certain countries or regions (e.g., how visible is literature from other 
regions in Google Scholar searches?). 


Conclusion 

We reported the geographic distribution, publication outlets, citations, data collection and analysis 
methods, and research strands of empirical research focusing on MOOCs in 2013-2015. We hope that this 
systematic analysis enables researchers to make better sense of the empirical literature on MOOCs and its 
direction and limitations. There are many possibilities for future research in this area. Future systematic 
reviews of the literature may focus on synthesizing knowledge on particular areas of interest, (e.g., 
completion and retention in MOOCs; learner motivations in MOOCs) or examining whether research 
methods used to understand MOOCs follow standard methods of inquiry or follow methods that take into 
advantage the digital nature of learning and teaching in this context. Further, future research may 
compare how papers published since this paper was written fit into the picture described herein and 
engage in further categorization and cross-tabulation of the literature. Finally, we hope that our results 
highlight the need for a critical reflection on the part of researchers as to why they study the topics that 
they study and to why they use the methods that they do. 
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